13 research outputs found
Virtuoso: Massive Multilingual Speech-Text Joint Semi-Supervised Learning for Text-To-Speech
This paper proposes Virtuoso, a massively multilingual speech-text joint
semi-supervised learning framework for text-to-speech synthesis (TTS) models.
Existing multilingual TTS typically supports tens of languages, which are a
small fraction of the thousands of languages in the world. One difficulty to
scale multilingual TTS to hundreds of languages is collecting high-quality
speech-text paired data in low-resource languages. This study extends Maestro,
a speech-text joint pretraining framework for automatic speech recognition
(ASR), to speech generation tasks. To train a TTS model from various types of
speech and text data, different training schemes are designed to handle
supervised (paired TTS and ASR data) and unsupervised (untranscribed speech and
unspoken text) datasets. Experimental evaluation shows that 1) multilingual TTS
models trained on Virtuoso can achieve significantly better naturalness and
intelligibility than baseline ones in seen languages, and 2) they can
synthesize reasonably intelligible and naturally sounding speech for unseen
languages where no high-quality paired TTS data is available.Comment: Submitted to ICASSP 202
Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages
We introduce the Universal Speech Model (USM), a single large model that
performs automatic speech recognition (ASR) across 100+ languages. This is
achieved by pre-training the encoder of the model on a large unlabeled
multilingual dataset of 12 million (M) hours spanning over 300 languages, and
fine-tuning on a smaller labeled dataset. We use multilingual pre-training with
random-projection quantization and speech-text modality matching to achieve
state-of-the-art performance on downstream multilingual ASR and speech-to-text
translation tasks. We also demonstrate that despite using a labeled training
set 1/7-th the size of that used for the Whisper model, our model exhibits
comparable or better performance on both in-domain and out-of-domain speech
recognition tasks across many languages.Comment: 20 pages, 7 figures, 8 table